status: draft
(contact chromeos-te@google.com)
author: jarbon@google.com
• Risk-based: Chrome OS has a large test surface area, spanning a customized browser, application manager user experience (UX), firmware, hardware, networking, user data synchronization, automatic updates, and customized physical hardware from Original Equipment Manufacturers (OEMs). To tackle this testing problem in a coherent manner, the test strategy is risk-based, meaning the team will focus on the most risky areas of the system first and work its way down the list, relying heavily on thorough unit tests and code quality from the development team to form a strong foundation of overall product quality.
• Hardware test matrix automation: Given the variety of hardware and continual OS builds, it is important to run tests on each continuous build across the entire hardware matrix to quickly identify any regressions and to aid in isolating issues of a particular software, hardware, or environmental set of dimensions (for example, a test fails only on HP hardware in a wireless configuration on browser buildX).
• Enable quick iteration: Chrome OS has an aggressive schedule, so there is a premium on identifying bugs as early as possible and on isolating the specific repro conditions. All tests are runnable on the local developer’s workstation to avoid bugs getting into the tree in the first place, and a large matrix of automation is executed to speed up the isolation of any regressions.
• Open tools and tests: Given the open source nature of ChromiumOS and the need for an OEM partner qualification, the test team makes every effort to ensure that the tools, tests, automation, and so on can be shared and executed externally.
• Chrome OS primary browser platform: The Chrome browser test team transitions to focusing on Chrome OS as a primary platform. Testability, automation, and so on are focused on Chrome OS first with other platforms a close second. This reflects the critical role of the Chrome browser in Chrome OS; it’s the only UX and everything in the OS and hardware is there to support its functionality. The Chrome browser quality bar must be higher on Chrome OS.
• Test delivers data: The goal of the test team isn’t, and cannot be, to ensure quality. Quality is a function of all disciplines on the project, including external OEMs, open-source projects, and so on. The test team’s goals are to mitigate risk, find as many issues and bugs as possible, and provide metrics and general evaluations of functional risk to the larger team. Test, dev, Program Manager (PM), and third parties have a voice and strong influence on quality for Chrome OS.
• Testability and the multiplier effect: Testability, especially for the Google apps teams, external third parties, and even internally has been an issue in the past. The test team partners with the accessibility, Android, and Webdriver teams to accelerate testability and make Chrome officially supported on Chrome OS. This improves test automation productivity within the team for the Google apps teams. This also has the ultimate benefit of making Chrome the ideal platform for testing other applications for third-party web pages.
The test team drives functional risk analysis with the following goals in mind:
• Ensure that the quality risks in the product are understood.
• Ensure that the test team is always focused on the highest return on investment (ROI) activities at any given time.
• Ensure that there is a framework for evaluating any new quality or risk data that comes in as the product evolves and new data arrives.
Risk analysis simply lists the set of all known features and capabilities of the product, and the test team evaluates the absolute inherent risk for each area given its frequency of exposure and likelihood of failure combined with the severity of consequences (to the user and the business) should it fail. Then, any mitigations (such as existing test cases, automation, testing via usage, OEM testing, and so on) are deducted from this known risk. All components are then stack-ranked by the net risk, and these areas are addressed through test case development, automation, and processes.
The bottom line is to know what is at risk in the product and to always apply the resources at hand to methodically reduce the risk in the product.
Every continual build receives the following testing in addition to dev unit tests via Buildbots.
• Smoke tests (P0 automation)
• Performance tests
Each day, the following testing occurs against the last known good (LKG) continual build:
• Manual verification of a set of functional acceptance tests (this can be limited to one type of hardware per day).
• Functional regression automation is executed.
• Continual testing of top web app tests picks up the daily rolling build (both automated and manual).
• Stress, long-running, and stability tests are conducted on a rolling basis. Re-run the tests on daily builds until no issues turn up, and then move to weekly runs.
• Continual manual exploratory testing and tour testing.
• AutoUpdate test automation.
Every “release candidate” build for all channels.
• Site compatibility: The Chrome browser test team verifies the top 100 sites on Chrome OS.
• Scenario verification: Verify scenarios of any demos that are commonly given (maybe a maximum of two to three demos) of the Chrome OS builds to partners or at conventions.
• P0 bug verification: Verify all fixed P0 bugs. Verify 80 percent of all P1 bugs filed since the period of the last release.
• Full stress and stability run: Conduct a stress and stability run.
• Chrome OS manual test cases: All Chrome OS manual test cases should be run (can be divided up between testers on different hardware).
Manual testing is important, especially early in the project when the UX and other features evolve quickly and when testability work and automation are underway. Manual testing is also critical because a core value of Chrome OS is simplicity, and the UX needs to be pleasant and intuitive for the user. Machines still cannot test this.
Automation is key to the long-term success and efficiency of the test team and to guard against regressions. As automation is enabled in the browser, much of manual testing in priority and ROI order is also automated.
The dev team is relatively large and has much more insight into component and CL-level technical details. We want dev to provide a rich set of unit and focused system tests that are added via Autotest.
The test team focuses on more end-to-end and integration test scenarios, focusing on user-exposed functionality, cross-component interaction, stability, and large-scale testing and reporting.
We should apply our learnings from the core Chrome browser team’s success with different “channels” for users with varying degrees of tolerance for pain and willingness to provide feedback. Several channels are maintained with increasing levels of confidence in quality. This mimics “flighting” of google.com properties and allows for some level of experimentation with real-world usage before a large deployment, mitigating risk across the product.
User input for quality is critically important. You need to facilitate making it easy for users to provide actionable feedback and to organize their data.
GoogleFeedback Extension: This extension allows users to provide point-and-click feedback on any URL, and it provides a dashboard for analysis and clustering of the feedback. The test team works to support the integration of GoogleFeedback into Chrome OS and to extend its reporting into the ChromeUX.
Known Bugs Extension / HUD: For trusted users with project tasks or context, it should be easy to file a bug on the OS and see existing bugs directly in the Chrome OS browser. Longer term, this project should develop into a more general purpose “heads-up display” for project and quality data.
Support rollout to Googlers aggressively, including supporting nonstandard hardware and spinup.
• Manual test cases: All manual test cases are stored in TestScribe. Later, folks work on a test case repository for code.google.com.
• Automated test cases: All are stored in the tree in Autotest format. Everything is versioned, shared, and right next to the code under test.
Given the large amount of data and need to disseminate it quickly, the test team invests in dedicated quality metrics dashboarding. This provides high-level, quality assessments (green and red lights), aggregates both manual and automated test execution results, and allows for deep dives into failure information.
It is important, especially early on, to support virtualization of the Chrome OS images. This reduces our dependence on physical hardware, speeds up imaging, supports regression testing in Selenium and WebDriver types of farms, and supports Chrome OS testing and development directly on workstations.
Generally speaking, performance is a core feature of Chrome OS, and as such, it is a large distributed project on the development team. The test team aims to aid in the execution, reporting, and trending of performance measurements in the lab, but not to focus on developing directed performance testing.
The test team focuses on building long-running test cases and executing them on physical hardware in the labs. Add fault injection via the underlying platform.
The test and dev teams have reached consensus that Autotest is the core test framework for test automation. Autotest is proven in the Linux community, used on several internal projects, and it is open source. Autotest also supports local and distributed execution. Autotest wraps all other functional test harnesses (such as WebDriver and other third-party test harnesses) so there is a unified interface to test execution, distribution, and reporting.
It is interesting to note that the core test tools team invests in adding Autotest support for Windows and Mac as well.
OEMs play a key role in qualification of Chrome OS builds. The test and dev teams work to deliver the relevant manual and automated test cases for OEMs to qualify builds and hardware on their end. The test team also works closely with top-tier OEMs to include hardware variants in the daily testing to catch any OEM-specific issues or regressions as early as possible.
A hardware lab is constructed to support a wide array of netbooks and devices with common services such as power, networking (wired and wireless), health dashboards, power management, and some specialized infrastructure for directed testing of things such as wireless. The lab machines are largely managed by HIVE infrastructure.
The test team builds out a farm of netbooks for test execution and reporting across the large matrix of hardware and software. These farms are distributed throughout MTV, KIR, and HYD locations to provide geo-local lab access and provide near round-the-clock test execution and debugging.
The core browser on Chrome OS is the Linux version of Chrome with a Chrome OS specialized UX and features. Much of the core rendering and functionality is the same, but there are significant differences between the core browser and the Chrome OS variant (such as pinned tabs, download manager, app launcher, platform control UX, wireless, and so on).
• Chrome OS is a primary platform for core Chrome browser testing (manual and automated).
• The core browser team determines which build of the browser to integrate (based on quality and Chrome OS features).
• For each Chrome OS release candidate, the core browser team performs the usual matrix of site (app) compatibility tests (top 300 sites) on Chrome OS (currently, there is only manual testing on live sites).
• The site (app) compatability tests are partially automated via WebDriver and integrated into builbots or regular lab runs, to provide “early warning” signals for any major Chrome OS specific regressions.
• A suite of manual tests focused on Chrome OS features of the browser and App Manager is developed and executed by a team of vendors.
• As APIs are implemented, the suite of manual Chrome OS tests are automated via vendors.
• Chrome OS Chromebot should have Linux and Chrome OS variants running with Chrome OS-specific functionality and not just web apps.
• Manual exploratory testing and tours should prove valuable in finding end user-focused issues around simplicity, functionality, usability, and so on.
The browser exposes many of the core UX and functional aspects of Chrome OS to the user. Much of the BrowserUX is not testable, or, it is testable only via lower-level IPC AutomationProxy interfaces that live outside of the browser. For Chrome OS, we work to unify the testing of web apps, Chrome UX, and functionality, and we work to kick off lower-level system tests. We also want to ensure that Chrome is the most testable browser for web apps, to encourage external web development teams to test first on the Chrome platform. To this end, the following is constructed:
• Port Selenium and WebDriver to Chrome OS: This is the core test framework driver for web apps today. The Chrome browser and OS test team are likely to pick up ownership of all Chrome-specific aspects of WebDriver moving forward to ensure a solid, testable interface for the apps teams and external testers.
• Expose Chrome UX and functionality via JavaScript DOM: This enables WebDriver tests to drive UX and functional aspects of Chrome as well. This functionality is exposed via the same methods as shutdown and sleep and the accessibility people (raman@) work on the ChromeViews.
• Higher-level scripting: Work with the WebDriver people to extend the basic WebDriver API into pure JavaScript and eventually into higher-order record/playback and parameterizable scripts (such as “Google Search <term>”). This speeds up internal and external test development, which, with WebDriver, still requires a fair amount of hunting for elements and maintenance issues when the UX changes quickly.
Chrome OS has a set of hardware requirements and variations from many OEMs. Directed testing is needed on the core OEM platforms to ensure tight integration between the physical hardware and Chrome OS. Specifically, tests are designed for:
• Power management: Line and battery-powered cycles, failures, power management of hardware components, and so on.
• Hardware malfunction: What does Chrome OS detect and how does it recover?
Q4 2009:
• Manual acceptance tests defined and running on continual builds
• Basic release qualification testing defined and executed for each major release
• Basic hardware labs spun up
• Automate E2E execution for continual builds on physical netbooks in the lab
• Hive support for virtual and physical machine imaging
• Port of WebDriver and Selenium to Chrome OS
• Some automated tests for top web apps
• Decide on test harness for dev and test
• Drive GoogleFeedback integration for Chrome OS
• Build out core test team, people, and processes
• Automated A/V tests
• Complete risk-based planning
• Manual UX testing plan
Q1 2010:
• Quality dashboard
• Automated auto update tests
• Support for automated performance testing in the lab
• Labs in HYD, KIR, and MTV all spun up and execute tests
• Chromebot for Linux and Chrome OS
• Testability support for major Chrome OS functions and UX
• Set of Chrome OS functional regression automation
• Add Chrome OS to web app regression Selenium farm
• Prototype of record and playback support for browser and UX testing
• ChromeSync E2E test case automation
• Stability and fault-injection testing
• Directed network testing
• Regular exploratory manual testing and tours to appease James J
Q2 2010:
• Risk mitigation via testing and automation
Q3 2010:
• Risk mitigation via testing and automation
Q4 2010:
• All risks are mitigated, everything is automated, there are no new issues, and there are no more UX or functional changes. The test team winds down.
• Test tech lead for Chrome OS platform
• Test tech lead for Chrome OS browser
• Core browser automation TechLead
• Dashboarding and metrics
• Manual acceptance test definition and execution
• Manual regression test definition and execution (and its team of vendors)
• Browser Appcompat, Core UX, and functionality (manual) and the team of vendors
• Audio and video
• Stability / fault injection
• Accessibility testing
• Hardware labs: KIR, HYD, and MTV
• Farm automation
• Hardware acceptance
• Overall direction and funding
• Risk analysis
• Hardware labs
• E2E farm automation
• Virtual and physical machine management infrastructure
• Heads-up Display (HUD)
• Manual acceptance tests
• Test results dashboard
• OEM hardware qualification tests
• Hardware usage and health dashboard
• Chrome OS manual/functional test plan